Performance Evaluation of Cloud Infrastructure using Complex Workloads

Infrastructure as a Service (IaaS) is a delivery model of cloud computing, which provides the ability to users to acquire and release resources according to their demand and pay according to their usage. Resources are provisioned from the cloud as Virtual Machines (VMs), many of which can be deployed on a single computing node, realizing a multi-tenancy model. While virtualization and multitenancy are two sources of workload-execution overhead that have been studied in the past, we still need a thorough, empirical investigation of the joint impact of these overheads, on workload execution. Additionally, commercial and private IaaS providers offer mechanisms that facilitate the lease and use of single infrastructure resources, but to execute multi-job workloads IaaS users still need to select adequate provisioning and allocation policies to instantiate resources and map computational jobs to them. Even though some studies on the policies employed in cloud environments already exist, current and potential IaaS users need deeper insight on the achieved performance and incurred cost of the used policies, derived through empirical investigation. In this work, we address these problems with the use of SkyMark, a performance analysis framework for IaaS clouds. SkyMark has three key features: first, it is designed to analyze IaaS deployments through a sequence of automated tests and the subsequent automated analysis of results. Second, it can analyze the impact of individual provisioning and allocation policies to the performance of the workload execution. Lastly, it is able to generate complex workloads, stressing any of the compute, memory and disk components. With the use of SkyMark, we first study the overheads that the cloud software stack imposes to the workload execution. Subsequently, we analyze the performance and cost of six provisioning and three allocation policies through experimentation in three IaaS environments, including Amazon EC2.